{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "vTOX0d3cmLQf"
   },
   "source": [
    "# EECS 498-007/598-005 Assignment 2-1: Linear Classifiers\n",
    "\n",
    "Before we start, please put your name and UMID in following format\n",
    "\n",
    ": Firstname LASTNAME, #00000000   //   e.g.) Justin JOHNSON, #12345678"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "tt7vq1h6mRto"
   },
   "source": [
    "**Your Answer:**   \n",
    "Your NAME, #XXXXXXXX"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "FrfeHl_-m4V-"
   },
   "source": [
    "## Setup Code\n",
    "Before getting started, we need to run some boilerplate code to set up our environment, same as Assignment 1. You'll need to rerun this setup code each time you start the notebook.\n",
    "\n",
    "First, run this cell load the autoreload extension. This allows us to edit .py source files, and re-import them into the notebook for a seamless editing and debugging experience."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 1027,
     "status": "ok",
     "timestamp": 1600042524722,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "VyQblYp0nEZq",
    "outputId": "1902f06e-d588-4ac5-97e9-299a8b2314f8"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The autoreload extension is already loaded. To reload it, use:\n",
      "  %reload_ext autoreload\n"
     ]
    }
   ],
   "source": [
    "%load_ext autoreload\n",
    "%autoreload 2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Q7ymI0aZ2W1b"
   },
   "source": [
    "### Google Colab Setup\n",
    "Next we need to run a few commands to set up our environment on Google Colab. If you are running this notebook on a local machine you can skip this section.\n",
    "\n",
    "Run the following cell to mount your Google Drive. Follow the link, sign in to your Google account (the same account you used to store this notebook!) and copy the authorization code into the text box that appears below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 344,
     "status": "ok",
     "timestamp": 1600042525194,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "c_LLpLyC2eau",
    "outputId": "5cc38a97-5396-4afc-ba83-400f486d22fd"
   },
   "outputs": [
    {
     "ename": "ModuleNotFoundError",
     "evalue": "No module named 'google.colab'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mModuleNotFoundError\u001b[0m                       Traceback (most recent call last)",
      "\u001b[1;32m~\\AppData\\Local\\Temp/ipykernel_4492/2045752835.py\u001b[0m in \u001b[0;36m<module>\u001b[1;34m\u001b[0m\n\u001b[1;32m----> 1\u001b[1;33m \u001b[1;32mfrom\u001b[0m \u001b[0mgoogle\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mcolab\u001b[0m \u001b[1;32mimport\u001b[0m \u001b[0mdrive\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0m\u001b[0;32m      2\u001b[0m \u001b[0mdrive\u001b[0m\u001b[1;33m.\u001b[0m\u001b[0mmount\u001b[0m\u001b[1;33m(\u001b[0m\u001b[1;34m'/content/drive'\u001b[0m\u001b[1;33m)\u001b[0m\u001b[1;33m\u001b[0m\u001b[1;33m\u001b[0m\u001b[0m\n\u001b[0;32m      3\u001b[0m \u001b[1;33m\u001b[0m\u001b[0m\n",
      "\u001b[1;31mModuleNotFoundError\u001b[0m: No module named 'google.colab'"
     ]
    }
   ],
   "source": [
    "from google.colab import drive\n",
    "drive.mount('/content/drive')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "mbq-UT8J2mnv"
   },
   "source": [
    "Now recall the path in your Google Drive where you uploaded this notebook, fill it in below. If everything is working correctly then running the folowing cell should print the filenames from the assignment:\n",
    "\n",
    "```\n",
    "['two_layer_net.ipynb', 'eecs598', 'two_layer_net.py', 'linear_classifier.py', 'linear_classifier.ipynb', 'a2_helpers.py']\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 303,
     "status": "ok",
     "timestamp": 1600042532597,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "WcrhTOZW243H",
    "outputId": "f294fd3e-3860-456f-b2f7-05bf799e60e2"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "\n",
    "# TODO: Fill in the Google Drive path where you uploaded the assignment\n",
    "# Example: If you create a 2020FA folder and put all the files under A2 folder, then '2020FA/A2'\n",
    "# GOOGLE_DRIVE_PATH_AFTER_MYDRIVE = '2020FA/A2'\n",
    "GOOGLE_DRIVE_PATH_AFTER_MYDRIVE = None\n",
    "GOOGLE_DRIVE_PATH = os.path.join('drive', 'My Drive', GOOGLE_DRIVE_PATH_AFTER_MYDRIVE)\n",
    "print(os.listdir(GOOGLE_DRIVE_PATH))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xegb0uDA232J"
   },
   "source": [
    "Once you have successfully mounted your Google Drive and located the path to this assignment, run th following cell to allow us to import from the `.py` files of this assignment. If it works correctly, it should print the message:\n",
    "\n",
    "```\n",
    "Hello from linear_classifier.py!\n",
    "```\n",
    "\n",
    "as well as the last edit time for the file `linear_classifier.py`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 787,
     "status": "ok",
     "timestamp": 1600042538544,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "AhGQF5sw3Fas",
    "outputId": "5e77649b-b04d-47a8-f67a-837516eb5874"
   },
   "outputs": [],
   "source": [
    "import sys\n",
    "sys.path.append(GOOGLE_DRIVE_PATH)\n",
    "\n",
    "import time, os\n",
    "os.environ[\"TZ\"] = \"US/Eastern\"\n",
    "time.tzset()\n",
    "\n",
    "from linear_classifier import hello_linear_classifier\n",
    "hello_linear_classifier()\n",
    "\n",
    "linear_classifier_path = os.path.join(GOOGLE_DRIVE_PATH, 'linear_classifier.py')\n",
    "linear_classifier_edit_time = time.ctime(os.path.getmtime(linear_classifier_path))\n",
    "print('linear_classifier.py last edited on %s' % linear_classifier_edit_time)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "ynKS05gJ4iBo"
   },
   "source": [
    "# Data preprocessing"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fN1SShPR4lJV"
   },
   "source": [
    "## Setup code\n",
    "Run some setup code for this notebook: Import some useful packages and increase the default figure size."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 255,
     "status": "ok",
     "timestamp": 1600042538923,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "VUCKw4Tl1ddj"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "import torch\n",
    "import torchvision\n",
    "import matplotlib.pyplot as plt\n",
    "import statistics\n",
    "import random\n",
    "import time\n",
    "import math\n",
    "%matplotlib inline\n",
    "\n",
    "\n",
    "plt.rcParams['figure.figsize'] = (10.0, 8.0)\n",
    "plt.rcParams['font.size'] = 16"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lhqpd2IN2O-K"
   },
   "source": [
    "Starting in this assignment, we will use the GPU to accelerate our computation. Run this cell to make sure you are using a GPU."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 436,
     "status": "ok",
     "timestamp": 1600042539504,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "SGDxdBIMRX6b",
    "outputId": "2fa0477a-31b2-4612-a154-b297589bd9e4"
   },
   "outputs": [],
   "source": [
    "if torch.cuda.is_available():\n",
    "  print('Good to go!')\n",
    "else:\n",
    "  print('Please set GPU via Edit -> Notebook Settings.')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "-Yv3zQYw5B3s"
   },
   "source": [
    "## Load the CIFAR-10 dataset\n",
    "Then, we will first load the CIFAR-10 dataset, same as knn. The utility function `eecs598.data.preprocess_cifar10()` returns the entire CIFAR-10 dataset as a set of six **Torch tensors**:\n",
    "\n",
    "- `X_train` contains all training images (real numbers in the range $[0, 1]$)\n",
    "- `y_train` contains all training labels (integers in the range $[0, 9]$)\n",
    "- `X_val` contains all validation images\n",
    "- `y_val` contains all validation labels\n",
    "- `X_test` contains all test images\n",
    "- `y_test` contains all test labels\n",
    "\n",
    "In this notebook we will use the **bias trick**: By adding an extra constant feature of ones to each image, we avoid the need to keep track of a bias vector; the bias will be encoded as the part of the weight matrix that interacts with the constant ones in the input.\n",
    "\n",
    "In the `two_layer_net.ipynb` notebook that follows this one, we will not use the bias trick.\n",
    "\n",
    "We can learn more about the `eecs598.data.preprocess_cifar10` function by invoking the `help` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 595
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 445,
     "status": "ok",
     "timestamp": 1600042539876,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "V2mFlFmQ1ddm",
    "outputId": "38262fbd-4b9c-45d2-f38a-83136aa24492"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "help(eecs598.data.preprocess_cifar10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "COCx2kM6XB3K"
   },
   "source": [
    "We can now run the `eecs598.data.preprocess` function to get our data:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 568
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 3040,
     "status": "ok",
     "timestamp": 1600042542853,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "r_BhZ_6_XB3K",
    "outputId": "92b4e51e-d0e0-43c7-e1f0-d3a116b1f06d"
   },
   "outputs": [],
   "source": [
    "# Invoke the above function to get our data. \n",
    "import eecs598\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "data_dict = eecs598.data.preprocess_cifar10(bias_trick=True, cuda=True, dtype=torch.float64)\n",
    "print('Train data shape: ', data_dict['X_train'].shape)\n",
    "print('Train labels shape: ', data_dict['y_train'].shape)\n",
    "print('Validation data shape: ', data_dict['X_val'].shape)\n",
    "print('Validation labels shape: ', data_dict['y_val'].shape)\n",
    "print('Test data shape: ', data_dict['X_test'].shape)\n",
    "print('Test labels shape: ', data_dict['y_test'].shape)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "0Lvdm4fm7iJC"
   },
   "source": [
    "# SVM Classifier\n",
    "\n",
    "In this section, you will:\n",
    "    \n",
    "- implement a fully-vectorized **loss function** for the SVM\n",
    "- implement the fully-vectorized expression for its **analytic gradient**\n",
    "- **check your implementation** using numerical gradient\n",
    "- use a validation set to **tune the learning rate and regularization** strength\n",
    "- **optimize** the loss function with **SGD**\n",
    "- **visualize** the final learned weights\n",
    "\n",
    "In Assignment 2, you SHOULD NOT use \".to()\" or \".cuda()\" in each implementation block. Otherwise, your implementation would gives you an error in Autograder end."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "xTI4qN7S9aTr"
   },
   "source": [
    "First, we will test the naive version of svm loss in `linear_classifier.py`. Let's first try the naive implementation of the loss we provided for you. You will get 9.000197. (Note: we've provided the `loss` part of the `svm_loss_naive` function, so you don't need to re-implement in `svm_loss_naive`. However, if your loss value doesn't match, then please report this to [Piazza](https://piazza.com/class/ke3a8m6u5wx647))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 22697,
     "status": "ok",
     "timestamp": 1600042563158,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "Hxzu2uZq9P8P",
    "outputId": "6c68b037-6ed6-4fbd-8dbf-faab64a41396"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import svm_loss_naive\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "# generate a random SVM weight tensor of small numbers\n",
    "W = torch.randn(3073, 10, dtype=data_dict['X_val'].dtype, device=data_dict['X_val'].device) * 0.0001 \n",
    "\n",
    "loss, _grad_ = svm_loss_naive(W, data_dict['X_val'], data_dict['y_val'], 0.000005)\n",
    "print('loss: %f' % (loss, ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "9LbRTXJ39Yp8"
   },
   "source": [
    "The `_grad_` returned from the function above is right now all zero. Derive and implement the gradient for the SVM cost function and implement it inline inside the function `svm_loss_naive`, by filing out the TODO blocks. You will find it helpful to interleave your new code inside the existing function.\n",
    "\n",
    "To check that you have implemented the gradient correctly, we will use **numeric gradient checking**: we will use a finite differences approach to numerically estimate the gradient of the forward pass, and compare this numeric gradient to the analytic gradient that you implemented.\n",
    "\n",
    "We have provided a function `eecs598.grad.grad_check_sparse` to help with numeric gradient checking. You can learn more about this function using the `help` command:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 323
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 22271,
     "status": "ok",
     "timestamp": 1600042563159,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "wzie6dahXB3Q",
    "outputId": "3d22fb55-f37d-476f-cf82-6fa2e077020e"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "help(eecs598.grad.grad_check_sparse)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "pKc4v6a8XB3T"
   },
   "source": [
    "Now run the following to perform numeric gradient checking on the gradients of your SVM loss. You should see relative errors less than `1e-5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 24726,
     "status": "ok",
     "timestamp": 1600042565991,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "o3sMha4i9p_V",
    "outputId": "728087f4-d83d-476c-d8b6-57e40378375e"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import svm_loss_naive\n",
    "\n",
    "# Once you've implemented the gradient, recompute it with the code below\n",
    "# and gradient check it with the function we provided for you\n",
    "\n",
    "# Use a random W and a minibatch of data from the val set for gradient checking\n",
    "# For numeric gradient checking it is a good idea to use 64-bit floating point\n",
    "# numbers for increased numeric precision; however when actually training models\n",
    "# we usually use 32-bit floating point numbers for increased speed.\n",
    "eecs598.reset_seed(0)\n",
    "W = 0.0001 * torch.randn(3073, 10, dtype=data_dict['X_val'].dtype, device=data_dict['X_val'].device)\n",
    "batch_size = 64\n",
    "X_batch = data_dict['X_val'][:batch_size]\n",
    "y_batch = data_dict['y_val'][:batch_size]\n",
    "\n",
    "# Compute the loss and its gradient at W.\n",
    "# YOUR_TURN: implement the gradient part of 'svm_loss_naive' function in \"linear_classifier.py\"\n",
    "_, grad = svm_loss_naive(W, X_batch, y_batch, reg=0.0)\n",
    "\n",
    "# Numerically compute the gradient along several randomly chosen dimensions, and\n",
    "# compare them with your analytically computed gradient. The numbers should\n",
    "# match almost exactly along all dimensions.\n",
    "f = lambda w: svm_loss_naive(w, X_batch, y_batch, reg=0.0)[0]\n",
    "grad_numerical = eecs598.grad.grad_check_sparse(f, W, grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "lSdsG-L292Ww"
   },
   "source": [
    "Let's do the gradient check once again with regularization turned on. (You didn't forget the regularization gradient, did you?)\n",
    "\n",
    "You should see relative errors less than `1e-5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 27345,
     "status": "ok",
     "timestamp": 1600042568998,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "bH6lXxVn9xZk",
    "outputId": "9de60bca-a304-42ef-9c71-896c7afd7e63"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import svm_loss_naive\n",
    "\n",
    "# Use a minibatch of data from the val set for gradient checking\n",
    "eecs598.reset_seed(0)\n",
    "W = 0.0001 * torch.randn(3073, 10, dtype=data_dict['X_val'].dtype, device=data_dict['X_val'].device)\n",
    "batch_size = 64\n",
    "X_batch = data_dict['X_val'][:batch_size]\n",
    "y_batch = data_dict['y_val'][:batch_size]\n",
    "\n",
    "# Compute the loss and its gradient at W.\n",
    "# YOUR_TURN: check your 'svm_loss_naive' implementation with different 'reg'\n",
    "_, grad = svm_loss_naive(W, X_batch, y_batch, reg=1e3) \n",
    "\n",
    "# Numerically compute the gradient along several randomly chosen dimensions, and\n",
    "# compare them with your analytically computed gradient. The numbers should\n",
    "# match almost exactly along all dimensions.\n",
    "f = lambda w: svm_loss_naive(w, X_batch, y_batch, reg=1e3)[0]\n",
    "grad_numerical = eecs598.grad.grad_check_sparse(f, W, grad)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "sc5Wtu-e-WlI"
   },
   "source": [
    "Now, let's implement vectorized version of SVM: `svm_loss_vectorized`. It should compute the same inputs and outputs as the naive version before, but it should involve **no explicit loops**.\n",
    "\n",
    "Let's first check the speed and performance bewteen the non-vectorized and the vectorized version. You should see a 15-120x speedup. PyTorch does some extra setup the first time you run CUDA code, so **you may need to run this cell more than once to see the desired speedup**.\n",
    "\n",
    "(Note: It may have some difference, but should be less than 1e-6)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 27538,
     "status": "ok",
     "timestamp": 1600042569595,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "pBLqLTAGo1Rs",
    "outputId": "89bba281-36e9-4f15-8f30-a0100c48ca57"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import svm_loss_naive, svm_loss_vectorized\n",
    "\n",
    "# Use random weights and a minibatch of val data for gradient checking\n",
    "eecs598.reset_seed(0)\n",
    "W = 0.0001 * torch.randn(3073, 10, dtype=data_dict['X_val'].dtype, device=data_dict['X_val'].device)\n",
    "X_batch = data_dict['X_val'][:128]\n",
    "y_batch = data_dict['y_val'][:128]\n",
    "reg = 0.000005\n",
    "\n",
    "# Run and time the naive version\n",
    "torch.cuda.synchronize()\n",
    "tic = time.time()\n",
    "loss_naive, grad_naive = svm_loss_naive(W, X_batch, y_batch, reg)\n",
    "torch.cuda.synchronize()\n",
    "toc = time.time()\n",
    "ms_naive = 1000.0 * (toc - tic)\n",
    "print('Naive loss: %e computed in %.2fms' % (loss_naive, ms_naive))\n",
    "\n",
    "# Run and time the vectorized version\n",
    "torch.cuda.synchronize()\n",
    "tic = time.time()\n",
    "# YOUR_TURN: implement the loss part of 'svm_loss_vectorized' function in \"linear_classifier.py\"\n",
    "loss_vec, _ = svm_loss_vectorized(W, X_batch, y_batch, reg)\n",
    "torch.cuda.synchronize()\n",
    "toc = time.time()\n",
    "ms_vec = 1000.0 * (toc - tic)\n",
    "print('Vectorized loss: %e computed in %.2fms' % (loss_vec, ms_vec))\n",
    "\n",
    "# The losses should match but your vectorized implementation should be much faster.\n",
    "print('Difference: %.2e' % (loss_naive - loss_vec))\n",
    "print('Speedup: %.2fX' % (ms_naive / ms_vec))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cRDPpAMl-0WD"
   },
   "source": [
    "Then, let's compute the gradient of the loss function. We can check the difference of gradient as well. (The error should be less than 1e-6)\n",
    "\n",
    "Now implement a vectorized version of the gradient computation in `svm_loss_vectorize` above. Run the cell below to compare the gradient of your naive and vectorized implementations. The difference between the gradients should be less than `1e-6`, and the vectorized version should run 15-120x faster.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 85
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 27496,
     "status": "ok",
     "timestamp": 1600042569954,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "3_SyWrTJ-OfX",
    "outputId": "333f5263-a25c-405a-95ab-257ca10064bf"
   },
   "outputs": [],
   "source": [
    "# The naive implementation and the vectorized implementation should match, but\n",
    "# the vectorized version should still be much faster.\n",
    "\n",
    "import eecs598\n",
    "from linear_classifier import svm_loss_naive, svm_loss_vectorized\n",
    "\n",
    "# Use random weights and a minibatch of val data for gradient checking\n",
    "eecs598.reset_seed(0)\n",
    "W = 0.0001 * torch.randn(3073, 10, dtype=data_dict['X_val'].dtype, device=data_dict['X_val'].device)\n",
    "X_batch = data_dict['X_val'][:128]\n",
    "y_batch = data_dict['y_val'][:128]\n",
    "reg = 0.000005\n",
    "\n",
    "# Run and time the naive version\n",
    "torch.cuda.synchronize()\n",
    "tic = time.time()\n",
    "_, grad_naive = svm_loss_naive(W, X_batch, y_batch, 0.000005)\n",
    "torch.cuda.synchronize()\n",
    "toc = time.time()\n",
    "ms_naive = 1000.0 * (toc - tic)\n",
    "print('Naive loss and gradient: computed in %.2fms' % ms_naive)\n",
    "\n",
    "# Run and time the vectorized version\n",
    "torch.cuda.synchronize()\n",
    "tic = time.time()\n",
    "# YOUR_TURN: implement the gradient part of 'svm_loss_vectorized' function in \"linear_classifier.py\"\n",
    "_, grad_vec = svm_loss_vectorized(W, X_batch, y_batch, 0.000005)\n",
    "torch.cuda.synchronize()\n",
    "toc = time.time()\n",
    "ms_vec = 1000.0 * (toc - tic)\n",
    "print('Vectorized loss and gradient: computed in %.2fms' % ms_vec)\n",
    "\n",
    "# The loss is a single number, so it is easy to compare the values computed\n",
    "# by the two implementations. The gradient on the other hand is a tensor, so\n",
    "# we use the Frobenius norm to compare them.\n",
    "grad_difference = torch.norm(grad_naive - grad_vec, p='fro')\n",
    "print('Gradient difference: %.2e' % grad_difference)\n",
    "print('Speedup: %.2fX' % (ms_naive / ms_vec))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "uU852IitCtrC"
   },
   "source": [
    "Now that we have an efficient vectorized implementation of the SVM loss and its gradient, we can implement a training pipeline for linear classifiers.\n",
    "\n",
    "Please complete the implementation of `train_linear_classifier` in `linear_classifer.py`.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "c6IL1_D9wCbF"
   },
   "source": [
    "Once you have implemented the training function, run the following cell to train a linear classifier using some default hyperparameters:\n",
    "\n",
    "(You should see a final loss close to 9.0, and your training loop should run in about two seconds)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 29022,
     "status": "ok",
     "timestamp": 1600042572063,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "QaEZkCe3-kOu",
    "outputId": "d0bb5f29-d597-4185-855e-beba0c8c325c"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import svm_loss_vectorized, train_linear_classifier\n",
    "\n",
    "# fix random seed before we perform this operation\n",
    "eecs598.reset_seed(0)\n",
    "\n",
    "torch.cuda.synchronize()\n",
    "tic = time.time()\n",
    "\n",
    "# YOUR_TURN: Implement how to construct the batch, \n",
    "#            and how to update the weight in 'train_linear_classifier'\n",
    "W, loss_hist = train_linear_classifier(svm_loss_vectorized, None, \n",
    "                                       data_dict['X_train'], \n",
    "                                       data_dict['y_train'], \n",
    "                                       learning_rate=3e-11, reg=2.5e4,\n",
    "                                       num_iters=1500, verbose=True)\n",
    "\n",
    "torch.cuda.synchronize()\n",
    "toc = time.time()\n",
    "print('That took %fs' % (toc - tic))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "n8mz9aXfDrsF"
   },
   "source": [
    "A useful debugging strategy is to plot the loss as a function of iteration number. In this case it seems our hyperparameters are not good, since the training loss is not decreasing very fast.\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 522
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 29018,
     "status": "ok",
     "timestamp": 1600042572490,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "JJ8GjaZS_MLe",
    "outputId": "5afa699a-6667-4c6f-982b-0ad610e6708a"
   },
   "outputs": [],
   "source": [
    "plt.plot(loss_hist, 'o')\n",
    "plt.xlabel('Iteration number')\n",
    "plt.ylabel('Loss value')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DRdfknKsE6F2"
   },
   "source": [
    "Then, let's move on to the prediction stage. We can evaluate the performance our trained model on both the training and validation set. You should see validation accuracy less than 20%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 28692,
     "status": "ok",
     "timestamp": 1600042572491,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "YfToPzce_OBH",
    "outputId": "be83e513-152e-4a60-e7f9-37e590ea6d3c"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import predict_linear_classifier\n",
    "\n",
    "# fix random seed before we perform this operation\n",
    "eecs598.reset_seed(0)\n",
    "\n",
    "# evaluate the performance on both the training and validation set\n",
    "# YOUR_TURN: Implement how to make a prediction with the trained weight \n",
    "#            in 'predict_linear_classifier'\n",
    "y_train_pred = predict_linear_classifier(W, data_dict['X_train'])\n",
    "train_acc = 100.0 * (data_dict['y_train'] == y_train_pred).double().mean().item()\n",
    "print('Training accuracy: %.2f%%' % train_acc)\n",
    "\n",
    "y_val_pred = predict_linear_classifier(W, data_dict['X_val'])\n",
    "val_acc = 100.0 * (data_dict['y_val'] == y_val_pred).double().mean().item()\n",
    "print('Validation accuracy: %.2f%%' % val_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "VWIyGnMOFOV8"
   },
   "source": [
    "Unfortunately, the performance of our initial model is quite bad. To find a better hyperparamters, we first modulized the functions that we've implemented as LinearSVM."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "taNmjt2wGJQr"
   },
   "source": [
    "Now, please use the validation set to tune hyperparameters (regularization strength and learning rate). You should experiment with different ranges for the learning rates and regularization strengths.\n",
    "\n",
    "To get full credit for the assignment your best model found through cross-validation should achieve an accuracy of at least 37% on the validation set.\n",
    "\n",
    "(Our best model got over 38.1% -- did you beat us?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 595
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 76733,
     "status": "ok",
     "timestamp": 1600042621103,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "oVMsJ9Ti_Ude",
    "outputId": "2f48ea8d-b1da-4e84-ad60-6ba99cd564a7"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import eecs598\n",
    "from linear_classifier import LinearSVM, svm_get_search_params, test_one_param_set\n",
    "\n",
    "# YOUR_TURN: find the best learning_rates and regularization_strengths combination\n",
    "#            in 'svm_get_search_params'\n",
    "learning_rates, regularization_strengths = svm_get_search_params()\n",
    "num_models = len(learning_rates) * len(regularization_strengths)\n",
    "\n",
    "####\n",
    "# It is okay to comment out the following conditions when you are working on svm_get_search_params.\n",
    "# But, please do not forget to reset back to the original setting once you are done.\n",
    "if num_models > 25:\n",
    "  raise Exception(\"Please do not test/submit more than 25 items at once\")\n",
    "elif num_models < 5:\n",
    "  raise Exception(\"Please present at least 5 parameter sets in your final ipynb\")\n",
    "####\n",
    "\n",
    "\n",
    "i = 0\n",
    "# results is dictionary mapping tuples of the form\n",
    "# (learning_rate, regularization_strength) to tuples of the form\n",
    "# (train_acc, val_acc). \n",
    "results = {}\n",
    "best_val = -1.0   # The highest validation accuracy that we have seen so far.\n",
    "best_svm_model = None # The LinearSVM object that achieved the highest validation rate.\n",
    "num_iters = 2000 # number of iterations\n",
    "\n",
    "for lr in learning_rates:\n",
    "  for reg in regularization_strengths:\n",
    "    i += 1\n",
    "    print('Training SVM %d / %d with learning_rate=%e and reg=%e'\n",
    "          % (i, num_models, lr, reg))\n",
    "    \n",
    "    eecs598.reset_seed(0)\n",
    "    # YOUR_TURN: implement a function that gives the trained model with \n",
    "    #            train/validation accuracies in 'test_one_param_set'\n",
    "    #            (note: this function will be used in Softmax Classifier section as well)\n",
    "    cand_svm_model, cand_train_acc, cand_val_acc = test_one_param_set(LinearSVM(), data_dict, lr, reg, num_iters)\n",
    "\n",
    "    if cand_val_acc > best_val:\n",
    "      best_val = cand_val_acc\n",
    "      best_svm_model = cand_svm_model # save the svm\n",
    "    results[(lr, reg)] = (cand_train_acc, cand_val_acc)\n",
    "\n",
    "\n",
    "# Print out results.\n",
    "for lr, reg in sorted(results):\n",
    "  train_acc, val_acc = results[(lr, reg)]\n",
    "  print('lr %e reg %e train accuracy: %f val accuracy: %f' % (\n",
    "         lr, reg, train_acc, val_acc))\n",
    "    \n",
    "print('best validation accuracy achieved during cross-validation: %f' % best_val)\n",
    "\n",
    "# save the best model\n",
    "path = os.path.join(GOOGLE_DRIVE_PATH, 'svm_best_model.pt')\n",
    "best_svm_model.save(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "wBbvJvMeGZ-7"
   },
   "source": [
    "Visualize the cross-validation results. You can use this as a debugging tool -- after examining the cross-validation results here, you may want to go back and rerun your cross-validation from above."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 713
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 76677,
     "status": "ok",
     "timestamp": 1600042621429,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "QbPffK9H_ZGj",
    "outputId": "fc56518b-406e-40fc-db07-667ceb817fe5"
   },
   "outputs": [],
   "source": [
    "x_scatter = [math.log10(x[0]) for x in results]\n",
    "y_scatter = [math.log10(x[1]) for x in results]\n",
    "\n",
    "# plot training accuracy\n",
    "marker_size = 100\n",
    "colors = [results[x][0] for x in results]\n",
    "plt.scatter(x_scatter, y_scatter, marker_size, c=colors, cmap='viridis')\n",
    "plt.colorbar()\n",
    "plt.xlabel('log learning rate')\n",
    "plt.ylabel('log regularization strength')\n",
    "plt.title('CIFAR-10 training accuracy')\n",
    "plt.gcf().set_size_inches(8, 5)\n",
    "plt.show()\n",
    "\n",
    "# plot validation accuracy\n",
    "colors = [results[x][1] for x in results] # default size of markers is 20\n",
    "plt.scatter(x_scatter, y_scatter, marker_size, c=colors, cmap='viridis')\n",
    "plt.colorbar()\n",
    "plt.xlabel('log learning rate')\n",
    "plt.ylabel('log regularization strength')\n",
    "plt.title('CIFAR-10 validation accuracy')\n",
    "plt.gcf().set_size_inches(8, 5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "MCMVzxquGf1O"
   },
   "source": [
    "Evaluate the best svm on test set. To get full credit for the assignment you should achieve a test-set accuracy above 35%.\n",
    "\n",
    "(Our best was over 39.1% -- did you beat us?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 76352,
     "status": "ok",
     "timestamp": 1600042621429,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "maJ7use3_soL",
    "outputId": "79a25bc9-9576-4316-aa61-558d92398101"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "y_test_pred = best_svm_model.predict(data_dict['X_test'])\n",
    "test_accuracy = torch.mean((data_dict['y_test'] == y_test_pred).double())\n",
    "print('linear SVM on raw pixels final test set accuracy: %f' % test_accuracy)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "I-QVIG4fGiqJ"
   },
   "source": [
    "Visualize the learned weights for each class. Depending on your choice of learning rate and regularization strength, these may or may not be nice to look at."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 385
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 76264,
     "status": "ok",
     "timestamp": 1600042621773,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "McLHYtFd_vSI",
    "outputId": "cf6c2474-a208-4947-cf5a-da4d08e03841"
   },
   "outputs": [],
   "source": [
    "w = best_svm_model.W[:-1,:] # strip out the bias\n",
    "w = w.reshape(3, 32, 32, 10)\n",
    "w = w.transpose(0, 2).transpose(1, 0)\n",
    "\n",
    "w_min, w_max = torch.min(w), torch.max(w)\n",
    "classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n",
    "for i in range(10):\n",
    "  plt.subplot(2, 5, i + 1)\n",
    "\n",
    "  # Rescale the weights to be between 0 and 255\n",
    "  wimg = 255.0 * (w[:, :, :, i].squeeze() - w_min) / (w_max - w_min)\n",
    "  plt.imshow(wimg.type(torch.uint8).cpu())\n",
    "  plt.axis('off')\n",
    "  plt.title(classes[i])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "DkuwyMY27RxS"
   },
   "source": [
    "# Softmax Classifier\n",
    "\n",
    "Similar to the SVM, you will:\n",
    "\n",
    "- implement a fully-vectorized **loss function** for the Softmax classifier\n",
    "- implement the fully-vectorized expression for its **analytic gradient**\n",
    "- **check your implementation** with numerical gradient\n",
    "- use a validation set to **tune the learning rate and regularization** strength\n",
    "- **optimize** the loss function with **SGD**\n",
    "- **visualize** the final learned weights\n",
    "\n",
    "As noted in the SVM section, you SHOULD NOT use \".to()\" or \".cuda()\" in each implementation block."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "hLJMVGtvIgo3"
   },
   "source": [
    "First, let's start from implementing the naive softmax loss function with nested loops in `softmax_loss_naive` function."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cER8fiSq7Ys-"
   },
   "source": [
    "As a sanity check to see whether we have implemented the loss correctly, run the softmax classifier with a small random weight matrix and no regularization. You should see loss near log(10) = 2.3"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 75605,
     "status": "ok",
     "timestamp": 1600042622065,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "V9q77O7F7VI6",
    "outputId": "47e913ca-0938-4b7d-8bec-703b0de6afcd"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import softmax_loss_naive\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "# Generate a random softmax weight tensor and use it to compute the loss.\n",
    "W = 0.0001 * torch.randn(3073, 10, dtype=data_dict['X_val'].dtype, device=data_dict['X_val'].device)\n",
    "\n",
    "X_batch = data_dict['X_val'][:128]\n",
    "y_batch = data_dict['y_val'][:128]\n",
    "\n",
    "# YOUR_TURN: Complete the implementation of softmax_loss_naive and implement \n",
    "# a (naive) version of the gradient that uses nested loops.\n",
    "loss, _ = softmax_loss_naive(W, X_batch, y_batch, reg=0.0)\n",
    "\n",
    "# As a rough sanity check, our loss should be something close to log(10.0).\n",
    "print('loss: %f' % loss)\n",
    "print('sanity check: %f' % (math.log(10.0)))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "5QJzUHl5I0HH"
   },
   "source": [
    "Next, we use gradient checking to debug the analytic gradient of our naive softmax loss function. If you've implemented the gradient correctly, you should see relative errors less than `1e-5`.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 78386,
     "status": "ok",
     "timestamp": 1600042625176,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "Lj6YpN3q1hVG",
    "outputId": "c66c4e7b-ae4d-4533-8921-bfa98d8e7d5f"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import softmax_loss_naive\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "W = 0.0001 * torch.randn(3073, 10, dtype=data_dict['X_val'].dtype, device=data_dict['X_val'].device)\n",
    "\n",
    "X_batch = data_dict['X_val'][:128]\n",
    "y_batch = data_dict['y_val'][:128]\n",
    "\n",
    "# YOUR_TURN: Complete the implementation of softmax_loss_naive and implement \n",
    "# a (naive) version of the gradient that uses nested loops.\n",
    "_, grad = softmax_loss_naive(W, X_batch, y_batch, reg=0.0)\n",
    "\n",
    "f = lambda w: softmax_loss_naive(w, X_batch, y_batch, reg=0.0)[0]\n",
    "eecs598.grad.grad_check_sparse(f, W, grad, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "cFcgeajBI-L3"
   },
   "source": [
    "Let's perform another gradient check with regularization enabled. Again you should see relative errors less than `1e-5`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 187
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 80765,
     "status": "ok",
     "timestamp": 1600042627921,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "Ik0i21sszZzg",
    "outputId": "d6752510-e1e8-46a6-845d-a8da92c468d3"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import softmax_loss_naive\n",
    "\n",
    "eecs598.reset_seed(128)\n",
    "W = 0.0001 * torch.randn(3073, 10, dtype=data_dict['X_val'].dtype, device=data_dict['X_val'].device)\n",
    "reg = 10.0\n",
    "\n",
    "X_batch = data_dict['X_val'][:128]\n",
    "y_batch = data_dict['y_val'][:128]\n",
    "\n",
    "# YOUR_TURN: Complete the gradient compuation part of softmax_loss_naive \n",
    "_, grad = softmax_loss_naive(W, X_batch, y_batch, reg)\n",
    "\n",
    "f = lambda w: softmax_loss_naive(w, X_batch, y_batch, reg)[0]\n",
    "eecs598.grad.grad_check_sparse(f, W, grad, 10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "JQgRzrdRJAm7"
   },
   "source": [
    "Then, let's move on to the vectorized form: `softmax_loss_vectorized`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "88xZ0rbLJGKV"
   },
   "source": [
    "Now that we have a naive implementation of the softmax loss function and its gradient, implement a vectorized version in softmax_loss_vectorized. The two versions should compute the same results, but the vectorized version should be much faster.\n",
    "\n",
    "The differences between the naive and vectorized losses and gradients should both be less than `1e-6`, and your vectorized implementation should be at least 20x faster than the naive implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 102
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 80384,
     "status": "ok",
     "timestamp": 1600042628071,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "lGNAe-oP1dds",
    "outputId": "b2bf5857-f19f-43ec-f00e-583ee8509277"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import softmax_loss_naive, softmax_loss_vectorized\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "W = 0.0001 * torch.randn(3073, 10, dtype=data_dict['X_val'].dtype, device=data_dict['X_val'].device)\n",
    "reg = 0.05\n",
    "\n",
    "X_batch = data_dict['X_val'][:128]\n",
    "y_batch = data_dict['y_val'][:128]\n",
    "\n",
    "# Run and time the naive version\n",
    "torch.cuda.synchronize()\n",
    "tic = time.time()\n",
    "loss_naive, grad_naive = softmax_loss_naive(W, X_batch, y_batch, reg)\n",
    "torch.cuda.synchronize()\n",
    "toc = time.time()\n",
    "ms_naive = 1000.0 * (toc - tic)\n",
    "print('naive loss: %e computed in %fs' % (loss_naive, ms_naive))\n",
    "\n",
    "# Run and time the vectorized version\n",
    "# YOUR_TURN: Complete the implementation of softmax_loss_vectorized\n",
    "torch.cuda.synchronize()\n",
    "tic = time.time()\n",
    "loss_vec, grad_vec = softmax_loss_vectorized(W, X_batch, y_batch, reg)\n",
    "torch.cuda.synchronize()\n",
    "toc = time.time()\n",
    "ms_vec = 1000.0 * (toc - tic)\n",
    "print('vectorized loss: %e computed in %fs' % (loss_vec, ms_vec))\n",
    "\n",
    "# we use the Frobenius norm to compare the two versions of the gradient.\n",
    "loss_diff = (loss_naive - loss_vec).abs().item()\n",
    "grad_diff = torch.norm(grad_naive - grad_vec, p='fro')\n",
    "print('Loss difference: %.2e' % loss_diff)\n",
    "print('Gradient difference: %.2e' % grad_diff)\n",
    "print('Speedup: %.2fX' % (ms_naive / ms_vec))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "bqZScXKyq6WB"
   },
   "source": [
    "Let's check that your implementation of the softmax loss is numerically stable.\n",
    "\n",
    "If either of the following print `nan` then you should double-check the numeric stability of your implementations."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 80385,
     "status": "ok",
     "timestamp": 1600042628472,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "bCyFPWxxq58R",
    "outputId": "a6ad7d89-c604-4b17-f669-c23567a4da79"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import softmax_loss_naive, softmax_loss_vectorized\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "device = data_dict['X_train'].device\n",
    "dtype = data_dict['X_train'].dtype\n",
    "D = data_dict['X_train'].shape[1]\n",
    "C = 10\n",
    "\n",
    "# YOUR_TURN??: train_linear_classifier should be same as what you've implemented in the SVM section\n",
    "W_ones = torch.ones(D, C, device=device, dtype=dtype)\n",
    "W, loss_hist = train_linear_classifier(softmax_loss_naive, W_ones, \n",
    "                                       data_dict['X_train'], \n",
    "                                       data_dict['y_train'], \n",
    "                                       learning_rate=1e-8, reg=2.5e4,\n",
    "                                       num_iters=1, verbose=True)\n",
    "\n",
    "\n",
    "W_ones = torch.ones(D, C, device=device, dtype=dtype)\n",
    "W, loss_hist = train_linear_classifier(softmax_loss_vectorized, W_ones, \n",
    "                                       data_dict['X_train'], \n",
    "                                       data_dict['y_train'], \n",
    "                                       learning_rate=1e-8, reg=2.5e4,\n",
    "                                       num_iters=1, verbose=True)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "kR4JGKoek8FB"
   },
   "source": [
    "Now lets train a softmax classifier with some default hyperparameters:\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 289
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 81950,
     "status": "ok",
     "timestamp": 1600042630391,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "Kqga1rvjk7b8",
    "outputId": "c5e0bace-afc3-476b-a57c-92a2540b3dae"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import softmax_loss_vectorized\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "\n",
    "torch.cuda.synchronize()\n",
    "tic = time.time()\n",
    "\n",
    "# YOUR_TURN??: train_linear_classifier should be same as what you've implemented in the SVM section\n",
    "W, loss_hist = train_linear_classifier(softmax_loss_vectorized, None, \n",
    "                                       data_dict['X_train'], \n",
    "                                       data_dict['y_train'], \n",
    "                                       learning_rate=1e-10, reg=2.5e4,\n",
    "                                       num_iters=1500, verbose=True)\n",
    "\n",
    "torch.cuda.synchronize()\n",
    "toc = time.time()\n",
    "print('That took %fs' % (toc - tic))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "QKjxCGwkorCc"
   },
   "source": [
    "Plot the loss curve:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 522
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 82110,
     "status": "ok",
     "timestamp": 1600042630932,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "K29x-DWNoujL",
    "outputId": "5d583690-69f6-427d-8f3c-953196eaaad3"
   },
   "outputs": [],
   "source": [
    "plt.plot(loss_hist, 'o')\n",
    "plt.xlabel('Iteration number')\n",
    "plt.ylabel('Loss value')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "7WvpBuJWSwfd"
   },
   "source": [
    "Let's compute the accuracy of current model. It should be less than 10%."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 51
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 80994,
     "status": "ok",
     "timestamp": 1600042630933,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "zb8kY2MjSvfH",
    "outputId": "20712234-bea7-4540-af2b-1d1d010c2caa"
   },
   "outputs": [],
   "source": [
    "import eecs598\n",
    "from linear_classifier import predict_linear_classifier\n",
    "\n",
    "eecs598.reset_seed(0)\n",
    "\n",
    "# evaluate the performance on both the training and validation set\n",
    "# YOUR_TURN??: predict_linear_classifier should be same as what you've implemented before, in the SVM section\n",
    "y_train_pred = predict_linear_classifier(W, data_dict['X_train'])\n",
    "train_acc = 100.0 * (data_dict['y_train'] == y_train_pred).double().mean().item()\n",
    "print('training accuracy: %.2f%%' % train_acc)\n",
    "y_val_pred = predict_linear_classifier(W, data_dict['X_val'])\n",
    "val_acc = 100.0 * (data_dict['y_val'] == y_val_pred).double().mean().item()\n",
    "print('validation accuracy: %.2f%%' % val_acc)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "IuV0BZvzJirI"
   },
   "source": [
    "Now use the validation set to tune hyperparameters (regularization strength and learning rate). You should experiment with different ranges for the learning rates and regularization strengths.\n",
    "\n",
    "To get full credit for the assignment, your best model found through cross-validation should achieve an accuracy above 0.37 on the validation set.\n",
    "\n",
    "(Our best model was above 39.8% -- did you beat us?)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 357
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 103159,
     "status": "ok",
     "timestamp": 1600042654105,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "68lmNVj31ddu",
    "outputId": "aacd16df-6ddf-4c3d-b854-337b4c16db83"
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import eecs598\n",
    "from linear_classifier import Softmax, softmax_get_search_params, test_one_param_set\n",
    "\n",
    "# YOUR_TURN: find the best learning_rates and regularization_strengths combination\n",
    "#            in 'softmax_get_search_params'\n",
    "learning_rates, regularization_strengths = softmax_get_search_params()\n",
    "num_models = len(learning_rates) * len(regularization_strengths)\n",
    "\n",
    "####\n",
    "# It is okay to comment out the following conditions when you are working on svm_get_search_params.\n",
    "# But, please do not forget to reset back to the original setting once you are done.\n",
    "if num_models > 25:\n",
    "  raise Exception(\"Please do not test/submit more than 25 items at once\")\n",
    "elif num_models < 5:\n",
    "  raise Exception(\"Please present at least 5 parameter sets in your final ipynb\")\n",
    "####\n",
    "\n",
    "\n",
    "i = 0\n",
    "# As before, store your cross-validation results in this dictionary.\n",
    "# The keys should be tuples of (learning_rate, regularization_strength) and\n",
    "# the values should be tuples (train_acc, val_acc)\n",
    "results = {}\n",
    "best_val = -1.0   # The highest validation accuracy that we have seen so far.\n",
    "best_softmax_model = None # The Softmax object that achieved the highest validation rate.\n",
    "num_iters = 2000 # number of iterations\n",
    "\n",
    "for lr in learning_rates:\n",
    "  for reg in regularization_strengths:\n",
    "    i += 1\n",
    "    print('Training Softmax %d / %d with learning_rate=%e and reg=%e'\n",
    "          % (i, num_models, lr, reg))\n",
    "    \n",
    "    eecs598.reset_seed(0)\n",
    "    cand_softmax_model, cand_train_acc, cand_val_acc = test_one_param_set(Softmax(), data_dict, lr, reg, num_iters)\n",
    "\n",
    "    if cand_val_acc > best_val:\n",
    "      best_val = cand_val_acc\n",
    "      best_softmax_model = cand_softmax_model # save the classifier\n",
    "    results[(lr, reg)] = (cand_train_acc, cand_val_acc)\n",
    "\n",
    "\n",
    "# Print out results.\n",
    "for lr, reg in sorted(results):\n",
    "  train_acc, val_acc = results[(lr, reg)]\n",
    "  print('lr %e reg %e train accuracy: %f val accuracy: %f' % (\n",
    "         lr, reg, train_acc, val_acc))\n",
    "    \n",
    "print('best validation accuracy achieved during cross-validation: %f' % best_val)\n",
    "\n",
    "# save the best model\n",
    "path = os.path.join(GOOGLE_DRIVE_PATH, 'softmax_best_model.pt')\n",
    "best_softmax_model.save(path)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "efougAmNCFLo"
   },
   "source": [
    "Run the following to visualize your cross-validation results:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 713
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 102558,
     "status": "ok",
     "timestamp": 1600042654440,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "IVhRe3-DBjPr",
    "outputId": "174e7c86-c8e7-444e-9fc0-2ba5ebeaac05"
   },
   "outputs": [],
   "source": [
    "x_scatter = [math.log10(x[0]) for x in results]\n",
    "y_scatter = [math.log10(x[1]) for x in results]\n",
    "\n",
    "# plot training accuracy\n",
    "marker_size = 100\n",
    "colors = [results[x][0] for x in results]\n",
    "plt.scatter(x_scatter, y_scatter, marker_size, c=colors, cmap='viridis')\n",
    "plt.colorbar()\n",
    "plt.xlabel('log learning rate')\n",
    "plt.ylabel('log regularization strength')\n",
    "plt.title('CIFAR-10 training accuracy')\n",
    "plt.gcf().set_size_inches(8, 5)\n",
    "plt.show()\n",
    "\n",
    "# plot validation accuracy\n",
    "colors = [results[x][1] for x in results] # default size of markers is 20\n",
    "plt.scatter(x_scatter, y_scatter, marker_size, c=colors, cmap='viridis')\n",
    "plt.colorbar()\n",
    "plt.xlabel('log learning rate')\n",
    "plt.ylabel('log regularization strength')\n",
    "plt.title('CIFAR-10 validation accuracy')\n",
    "plt.gcf().set_size_inches(8, 5)\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "fbOlUcv6J7MM"
   },
   "source": [
    "Them, evaluate the performance of your best model on test set. To get full credit for this assignment you should achieve a test-set accuracy above 0.36.\n",
    "\n",
    "(Our best was just around 39.9% -- did you beat us?)\n",
    "\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 34
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 101868,
     "status": "ok",
     "timestamp": 1600042654843,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "-wxkVdB-1ddx",
    "outputId": "f5ec9727-12f1-4efc-e333-e373b8a79f93"
   },
   "outputs": [],
   "source": [
    "y_test_pred = best_softmax_model.predict(data_dict['X_test'])\n",
    "test_accuracy = torch.mean((data_dict['y_test'] == y_test_pred).double())\n",
    "print('softmax on raw pixels final test set accuracy: %f' % (test_accuracy, ))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "colab_type": "text",
    "id": "Joo4RbeoKECC"
   },
   "source": [
    "Finally, let's visualize the learned weights for each class"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 385
    },
    "colab_type": "code",
    "executionInfo": {
     "elapsed": 100881,
     "status": "ok",
     "timestamp": 1600042655002,
     "user": {
      "displayName": "Yunseok Jang",
      "photoUrl": "https://lh3.googleusercontent.com/a-/AOh14Gji2utsrQJWXntm3ishdCA23wmdDA4QyRS8UrqQsEQ=s64",
      "userId": "10051210866960976186"
     },
     "user_tz": 240
    },
    "id": "XDfxI7mR1ddz",
    "outputId": "30116f98-b621-4c29-b7b7-710f8d1626bb"
   },
   "outputs": [],
   "source": [
    "w = best_softmax_model.W[:-1,:] # strip out the bias\n",
    "w = w.reshape(3, 32, 32, 10)\n",
    "w = w.transpose(0, 2).transpose(1, 0)\n",
    "\n",
    "w_min, w_max = torch.min(w), torch.max(w)\n",
    "\n",
    "classes = ['plane', 'car', 'bird', 'cat', 'deer', 'dog', 'frog', 'horse', 'ship', 'truck']\n",
    "for i in range(10):\n",
    "  plt.subplot(2, 5, i + 1)\n",
    "\n",
    "  # Rescale the weights to be between 0 and 255\n",
    "  wimg = 255.0 * (w[:, :, :, i].squeeze() - w_min) / (w_max - w_min)\n",
    "  plt.imshow(wimg.type(torch.uint8).cpu())\n",
    "  plt.axis('off')\n",
    "  plt.title(classes[i])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {},
    "colab_type": "code",
    "id": "GsmqO2p2XYhi"
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "accelerator": "GPU",
  "colab": {
   "collapsed_sections": [
    "FrfeHl_-m4V-",
    "Q7ymI0aZ2W1b",
    "ynKS05gJ4iBo",
    "fN1SShPR4lJV",
    "-Yv3zQYw5B3s",
    "0Lvdm4fm7iJC",
    "DkuwyMY27RxS"
   ],
   "name": "linear_classifier.ipynb",
   "provenance": [],
   "toc_visible": true
  },
  "interpreter": {
   "hash": "e7b9717bf57cd113384d631e3a4b00c383511bed8b2957a7bb28a8c742f810a2"
  },
  "kernelspec": {
   "display_name": "Python 3.7.11 64-bit ('py37': conda)",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.11"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
